Skip to content

Conversation

DenzelPenzel
Copy link
Contributor

Description

Add some check in CI that would verify if test has a corresponding issue and if that issue exists. That verification could happen whenever some flaky test is added to the zombienet-flaky-tests.

@DenzelPenzel DenzelPenzel requested review from a team as code owners October 15, 2025 22:03
@DenzelPenzel DenzelPenzel added R0-no-crate-publish-required The change does not require any crates to be re-published. T10-tests This PR/Issue is related to tests. labels Oct 15, 2025
@DenzelPenzel DenzelPenzel force-pushed the flaky-validation-tests branch from 071ac7d to 45c98c8 Compare October 16, 2025 09:18
@paritytech-review-bot paritytech-review-bot bot requested a review from a team October 16, 2025 09:19
@DenzelPenzel DenzelPenzel force-pushed the flaky-validation-tests branch 2 times, most recently from fd2bbc8 to 335530e Compare October 16, 2025 09:24
@paritytech-workflow-stopper
Copy link

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/18556622090
Failed job name: test-linux-stable

Copy link
Contributor

@lrubasze lrubasze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using gh cli? Just to simplify the script a bit.
It is widely used in the CI.
eg.

gh issue comment $PR --edit-last -F msg.txt || gh issue comment $PR -F msg.txt

Eg. we could get the issue state with this:

gh issue view 8944 --json state,title

@pepoviola
Copy link
Contributor

gh issue view 8944 --json state,title

Yes, I think is a better approach. We can just query the state and ensure is OPEN

something like

if [[ $(gh issue view 1243 --json state  &>1 | jq -r .state) == "OPEN" ]];then echo pass; else echo err;fi

thanks @DenzelPenzel / @lrubasze 🙌

@DenzelPenzel DenzelPenzel force-pushed the flaky-validation-tests branch 3 times, most recently from f9f398d to a37c1ea Compare October 17, 2025 14:20
@DenzelPenzel DenzelPenzel force-pushed the flaky-validation-tests branch from a37c1ea to 150e8b3 Compare October 17, 2025 14:21
Copy link
Contributor

@lrubasze lrubasze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Just noticed that PR numbers are treated like issue numbers.
See this:

❯ gh issue view 10034 --json state,title,url,id,stateReason,number
{
  "id": "PR_kwDOKDT1Ss6t-Z9p",
  "number": 10034,
  "state": "OPEN",
  "stateReason": "",
  "title": "feat: add validation for zombienet flaky tests",
  "url": "https://github.com/paritytech/polkadot-sdk/pull/10034"
}

vs

❯ gh issue view 9980 --json state,title,url,id,stateReason,number
{
  "id": "I_kwDOKDT1Ss7QkIP6",
  "number": 9980,
  "state": "OPEN",
  "stateReason": "",
  "title": "Fix zombienet-polkadot-functional-0014-chunk-fetching-network-compatibility",
  "url": "https://github.com/paritytech/polkadot-sdk/issues/9980"
}

Not sure. Maybe we could just ignore it. Alternatively we could detect it it by checking the url or id in order to distinguish if it is an issue or not.

@lrubasze
Copy link
Contributor

I think test zombienet-polkadot-smoke-0003-deregister-register-validator needs to have an issue, otherwise the job will be failing:
https://github.com/paritytech/polkadot-sdk/blob/150e8b399afb81cd466a6837d0f5587b62b32fd0/.github/zombienet-flaky-tests#L2C1-L2C60

@pepoviola
Copy link
Contributor

LGTM

Just noticed that PR numbers are treated like issue numbers. See this:

❯ gh issue view 10034 --json state,title,url,id,stateReason,number
{
  "id": "PR_kwDOKDT1Ss6t-Z9p",
  "number": 10034,
  "state": "OPEN",
  "stateReason": "",
  "title": "feat: add validation for zombienet flaky tests",
  "url": "https://github.com/paritytech/polkadot-sdk/pull/10034"
}

vs

❯ gh issue view 9980 --json state,title,url,id,stateReason,number
{
  "id": "I_kwDOKDT1Ss7QkIP6",
  "number": 9980,
  "state": "OPEN",
  "stateReason": "",
  "title": "Fix zombienet-polkadot-functional-0014-chunk-fetching-network-compatibility",
  "url": "https://github.com/paritytech/polkadot-sdk/issues/9980"
}

Not sure. Maybe we could just ignore it. Alternatively we could detect it it by checking the url or id in order to distinguish if it is an issue or not.

Yes, I think from GH point of view are two variant (issue/pr) and the number will not be repeated. We should add a regex to check the url is for an issue. 👍

Thx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

R0-no-crate-publish-required The change does not require any crates to be re-published. T10-tests This PR/Issue is related to tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants